Multiplexing optical assembly with a high resolution inset

ABSTRACT

A display assembly presented herein includes an inset display, a peripheral display, and a multiplexing optical assembly (MOA). The inset display has a first resolution and emits image light of a first polarization. The peripheral display has a second resolution and emits image light of a second polarization. The MOA receives the image light of the first polarization and the image light of the second polarization. The MOA then transforms the image light of the first polarization into a first portion of image light of a third polarization, and transforms the image light of the second polarization into a second portion of image light of the third polarization. The MOA directs the first portion of image light and the second portion of image light toward an eye-box. The display assembly can be implemented as a component of a head-mounted display of an artificial reality system.

BACKGROUND

The present disclosure generally relates to displaying content to a userof an artificial reality system, and specifically relates to amultiplexing optical assembly with a high resolution inset display.

Conventional displays present images at a constant resolution. Incontrast, resolution varies across a retina of a human eye. Though theeye receives data from a field of about 200 degrees, the acuity overmost of that range is poor. In fact, the light must fall on the fovea toform high resolution images, and that limits the acute vision angle toabout 15 degrees. In head-mounted displays (HMDs), at any given time,only a small portion of the image light emitted from the display isactually imaged onto the fovea. The remaining image light that is imagedonto retina is imaged at other areas that are not capable of perceivingthe high resolution in the emitted image light. Accordingly, some of theresources (e.g., power, memory, processing time, etc.) that went intogenerating the high resolution image being viewed by the user is wastedas the user is not able to perceive the portion of the image lightimaged outside the fovea at its full resolution.

SUMMARY

A display assembly presented herein includes an inset display, aperipheral display, and a multiplexing optical assembly (MOA). The insetdisplay has a first resolution and is configured to emit image light ofa first polarization. The peripheral display has a second resolution andis configured to emit image light of a second polarization. In someembodiments, the first resolution of the inset display is higher thanthe second resolution of the peripheral display. The inset displayrepresents a high resolution display positioned along an optical axis ofthe MOA. The inset display covers a first narrow field-of-view (FOV)around the optical axis corresponding to a foveal region of an eye. Theperipheral display spans throughout a second FOV wider than the firstFOV covering a retinal region of the eye outside the fovea. In someembodiments, the inset display is positioned in parallel with theperipheral display. The MOA receives image light of the firstpolarization emitted from the inset display and image light of thesecond polarization emitted from the peripheral display. The MOA thentransforms the image light of the first polarization into a firstportion of image light having a third polarization. The MOA alsotransforms the image light of the second polarization into a secondportion of image light having the third polarization. The MOA directsthe first portion of image light and the second portion of image lighttoward an eye-box where the eye is located.

A head-mounted display (HMD) can further integrate the display assembly.The HMD displays content to a user. The HMD may be part of an artificialreality system. The HMD includes an electronic display and an opticalassembly. The electronic display is configured to emit image light. Theelectronic display may include the inset display and the peripheraldisplay of the display assembly. The optical assembly is configured todirect the image light to an eye-box of the HMD corresponding to alocation of a user's eye. The optical assembly may include the MOA ofthe display assembly.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a head-mounted display (HMD), in accordance withan embodiment.

FIG. 2 is a cross section of a front rigid body of the HMD in FIG. 1, inaccordance with an embodiment.

FIG. 3 is a block diagram illustrating a detailed view of modules withina display assembly, which may be part of the HMD in FIG. 1, inaccordance with an embodiment.

FIG. 4 is a detailed view of cross section of the display assembly ofFIG. 3 including an inset display, a peripheral display and amultiplexing optical assembly, in accordance with an embodiment.

FIG. 4A shows propagation of image light emitted from an inset display,in accordance with an embodiment.

FIG. 4B shows superposition of display paths from an inset display and aperipheral display, in accordance with an embodiment.

FIG. 4C shows superposition of display paths from the inset display andthe peripheral display for a field angle larger than that in FIG. 4B, inaccordance with an embodiment.

FIG. 5 is a detailed view of cross section of the display assembly ofFIG. 3 having a multiplexing optical assembly based on at least onebirefringent lens, in accordance with an embodiment.

FIG. 6 is a block diagram of a HMD system in which a console operates,in accordance with an embodiment.

The figures depict embodiments of the present disclosure for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles, or benefits touted, of the disclosure described herein.

DETAILED DESCRIPTION

Embodiments of the present disclosure may include or be implemented inconjunction with an artificial reality system. Artificial reality is aform of reality that has been adjusted in some manner beforepresentation to a user, which may include, e.g., a virtual reality (VR),an augmented reality (AR), a mixed reality (MR), a hybrid reality, orsome combination and/or derivatives thereof. Artificial reality contentmay include completely generated content or generated content combinedwith captured (e.g., real-world) content. The artificial reality contentmay include video, audio, haptic feedback, or some combination thereof,and any of which may be presented in a single channel or in multiplechannels (such as stereo video that produces a three-dimensional effectto the viewer). Additionally, in some embodiments, artificial realitymay also be associated with applications, products, accessories,services, or some combination thereof, that are used to, e.g., createcontent in an artificial reality and/or are otherwise used in (e.g.,perform activities in) an artificial reality. The artificial realitysystem that provides the artificial reality content may be implementedon various platforms, including a HMD connected to a host computersystem, a standalone HMD, a mobile device or computing system, or anyother hardware platform capable of providing artificial reality contentto one or more viewers.

A display assembly presented herein includes an inset display, aperipheral display, and a multiplexing optical assembly (MOA). The MOAincludes two optical elements in optical series that form a pancakelens. The MOA folds light of a first polarization and transmits light ofa second polarization (e.g., polarization orthogonal to the firstpolarization). The inset display is a high resolution display that ispositioned along an optical axis of the MOA. The inset display emitsimage light of the first polarization. The peripheral display has alower pixel resolution than the inset display, and emits image light atthe second polarization such that the image light of the secondpolarization is not folded by the MOA.

In some embodiments, the display assembly is incorporated into a HMD.The HMD displays content to a user. The HMD may be part of an artificialreality system. The HMD includes an electronic display and an opticalassembly. The electronic display is configured to emit image light. Theelectronic display may include the inset display and the peripheraldisplay of the display assembly. The optical assembly is configured todirect the image light to an eye-box of the HMD corresponding to alocation of a user's eye. The optical assembly may include the MOA ofthe display assembly. The MOA combines image light emitted from theinset display and image light emitted from the peripheral display, anddirects the combined light to the eye-box. In this way, a widefield-of-view (FOV) is achieved while high resolution images areprovided in a central region of the FOV, i.e., in a foveal region of aretina of the user's eye.

FIG. 1 is a diagram of a HMD 100, in accordance with an embodiment. TheHMD 100 may be part of an artificial reality system. In embodiments thatdescribe AR system and/or a MR system, portions of a front side 102 ofthe HMD 100 are at least partially transparent in the visible band (˜380nm to 750 nm), and portions of the HMD 100 that are between the frontside 102 of the HMD 100 and an eye of the user are at least partiallytransparent (e.g., a partially transparent electronic display). The HMD100 includes a front rigid body 105, a band 110, and a reference point115. In some embodiments, the HMD 100 may also include a depth cameraassembly (DCA) configured to determine depth information of a local areasurrounding some or all of the HMD 100. The HMD 100 may also include animaging aperture 120 and an illumination aperture 125, and anillumination source of the DCA emits light (e.g., a structured lightpattern) through the illumination aperture 125. An imaging device of theDCA captures light from the illumination source that is reflected fromthe local area through the imaging aperture 120.

The front rigid body 105 includes one or more electronic displayelements (not shown in FIG. 1), one or more integrated eye trackingsystems (not shown in FIG. 1), an Inertial Measurement Unit (IMU) 130,one or more position sensors 135, and the reference point 115. In theembodiment shown by FIG. 1, the position sensors 135 are located withinthe IMU 130, and neither the IMU 130 nor the position sensors 135 arevisible to a user of the HMD 100. The IMU 130 is an electronic devicethat generates IMU data based on measurement signals received from oneor more of the position sensors 135. A position sensor 135 generates oneor more measurement signals in response to motion of the HMD 100.Examples of position sensors 135 include: one or more accelerometers,one or more gyroscopes, one or more magnetometers, another suitable typeof sensor that detects motion, a type of sensor used for errorcorrection of the IMU 130, or some combination thereof. The positionsensors 135 may be located external to the IMU 130, internal to the IMU130, or some combination thereof.

FIG. 2 is a cross section 200 of the front rigid body 105 of the HMD 100shown in FIG. 1. As shown in FIG. 2, the front rigid body 105 includesan inset display 210, a peripheral display 220, and an optical assembly230 that together provide compound image light to an eye-box 235. Theeye-box 235 is a region in space that is occupied by a user's eye 240.For purposes of illustration, FIG. 2 shows a cross section 200associated with a single eye 240, but another optical assembly 230,separate from the optical assembly 230, provides altered image light toanother eye of the user.

The inset display 210 is a high resolution display that emits imagelight covering a first FOV around a foveal region of the user's eye 240.In some embodiments, the first FOV spans between −10 degrees and +10degrees. The inset display 210 may emit polarized image light. In someembodiments, the inset display 210 directly generates the polarizedimage light. Alternatively, the inset display 210 includes a displaysurface and a polarizer (not shown in FIG. 2). The display surface ofthe inset display 210 is configured to emit image light, and thepolarizer polarizes the image light to form the polarized image light.In some embodiments, the inset display 210 includes an optical elementthat adjusts the focus of the generated image light. The inset display210 displays portions of high resolution images to the user inaccordance with data received from a console (not shown in FIG. 1). Invarious embodiments, the inset display 210 may comprise a singleelectronic display or multiple electronic displays (e.g., a display foreach eye of a user). Examples of the inset display 210 include: a liquidcrystal display (LCD), an organic light emitting diode (OLED) display,an inorganic light emitting diode (ILED) display, an active-matrixorganic light-emitting diode (AMOLED) display, a transparent organiclight emitting diode (TOLED) display, some other display, a projector,or some combination thereof. The inset display 210 may also include anaperture, a Fresnel lens, a convex lens, a concave lens, a diffractiveelement, a waveguide, a filter, a polarizer, a diffuser, a fiber taper,a reflective surface, a polarizing reflective surface, or any othersuitable optical element that affects the image light emitted from theelectronic display. In some embodiments, the inset display 210 may haveone or more coatings, such as anti-reflective coatings.

The peripheral display 220 is a lower resolution display that emitsimage light covering a second FOV in a retinal (peripheral) region ofthe user's eye 240 outside the foveal region. In general, the second FOVis much wider than the first FOV covered by the inset display 215. Theperipheral display 220 may generate image light having polarizationdifferent than polarization of the image light emitted from the insetdisplay 210. In some embodiments, the peripheral display 220 includes anoptical element that adjusts the focus of the generated image light. Theperipheral display 220 displays portions of lower resolution images tothe user in accordance with data received from the console. In variousembodiments, the peripheral display 220 may comprise a single electronicdisplay or multiple electronic displays (e.g., a display for each eye ofa user). Examples of the peripheral display 220 include: a LCD, an OLEDdisplay, an ILED display, an AMOLED display, a TOLED display, some otherdisplay, a projector, or some combination thereof. The peripheraldisplay 220 may also include an aperture, a Fresnel lens, a convex lens,a concave lens, a diffractive element, a waveguide, a filter, apolarizer, a diffuser, a fiber taper, a reflective surface, a polarizingreflective surface, or any other suitable optical element that affectsthe image light emitted from the electronic display. In someembodiments, the peripheral display 220 may have one or more coatings,such as anti-reflective coatings.

The optical assembly 230 receives image light from the inset display 210and the peripheral display 220, combines (multiplexes) the receivedimage light and directs the combined image light to the eye-box 235 ofthe user's eye 240. The optical assembly 230 also magnifies the receivedimage light, corrects optical aberrations associated with the imagelight, and the corrected image light is presented to a user of the HMD100. At least one optical element of the optical assembly 230 may be anaperture, a Fresnel lens, a refractive lens, a reflective surface, adiffractive element, a waveguide, a filter, or any other suitableoptical element that affects image light emitted from the inset display210 and image light emitted from the peripheral display 220. Moreover,the optical assembly 230 may include combinations of different opticalelements. In some embodiments, one or more of the optical elements inthe optical assembly 230 may have one or more coatings, such asanti-reflective coatings, dichroic coatings, etc. Magnification of theimage light by the optical assembly 230 allows elements of the insetdisplay 210 and the peripheral display 220 to be physically smaller,weigh less, and consume less power than larger displays. Additionally,magnification may increase a FOV of the displayed media. For example,the FOV of the displayed media is such that the displayed media ispresented using almost all (e.g., 110 degrees diagonal), and in somecases all, of the user's FOV. In some embodiments, the optical assembly230 is designed so its effective focal length is larger than the spacingto the peripheral display 220, which magnifies the image light projectedby the peripheral display 220. Additionally, in some embodiments, theamount of magnification may be adjusted by adding or removing opticalelements. More details about embodiments of the inset display 210, theperipheral display 220 and the optical assembly 230 are described inconjunction with FIG. 3 and FIG. 4.

In some embodiments, the front rigid body 105 further comprises an eyetracking system (not shown in FIG. 1) that determines eye trackinginformation for the user's eye 240. The determined eye trackinginformation may comprise information about a position (includingorientation) of the user's eye 240 in the eye-box 235, i.e., informationabout an angle of an eye-gaze. In one embodiment, the eye trackingsystem illuminates the user's eye 240 with structured light. The eyetracking system can use locations of the reflected structured light in acaptured image to determine the position of the user's eye 240. Inanother embodiment, the eye tracking system determines the position ofthe user's eye 240 based on magnitudes of image light captured over aplurality of time instants.

In some embodiments, the front rigid body 105 further comprises avarifocal module (not shown in FIG. 2). The varifocal module may adjustfocus of one or more images displayed on the inset display 210 and/orthe peripheral display 220, based on the eye tracking informationobtained from the eye tracking system. In one embodiment, the varifocalmodule adjusts focus of the displayed images and mitigatesvergence-accommodation conflict by adjusting a focal distance of theoptical assembly 230 based on the determined eye tracking information.In another embodiment, the varifocal module adjusts focus of thedisplayed images by performing foveated rendering of the one or moreimages based on the determined eye tracking information.

FIG. 3 is a high-level block diagram illustrating a detailed view ofmodules within a display assembly 300, in accordance with an embodiment.In some embodiments, the display assembly 300 is a component of thefront rigid body 205 of the HMD 100. In alternate embodiments, thedisplay assembly 300 is part of some other HMD, or other system thatgenerates images at high resolution around a center of a wide FOV(optical axis).

The display assembly 300 includes a combined display element 305 thatfurther includes at least one high inset display 310 and at least onelower resolution peripheral display 315, a MOA 320, and a controller325. The configuration and operation of the inset display 310, theperipheral display 315, and the MOA 320 are similar to the inset display215, the peripheral display 220, and the optical assembly 230 of thefront rigid body 105 of FIG. 2, respectively.

The display assembly 300 displays composite content to the user, e.g.,in accordance with data received from a console (not shown in FIG. 3).Composite content includes an inset region and a peripheral (orbackground) region. The inset region includes a high resolution insetportion of an image. For example, the high resolution inset portion hasresolution corresponding to a resolution of a fovea region of a humaneye. The peripheral region has a resolution corresponding to a non-fovearegion of a human eye. In various embodiments, the display assembly 300may comprise at least two electronic displays for each eye of a user,for example the inset display 310 and the peripheral display 315.Examples of the electronic displays include: a LCD, an OLED display, anAMOLED display, some other display, or some combination thereof.

The inset display 310 displays the inset region portion of the compositecontent. The inset display 310 has at least a resolution capable ofdisplaying the high resolution inset portion of the image at its fullresolution. In one embodiment, an optic of the inset display 310features an effective focal length (EFL) of approximately 20 mm; a pixelpitch of a display surface of the inset display 310 is approximately 12μm; and an instantaneous FOV (spatial resolution) is approximately 2.0arcminute. In some embodiments, along the periphery of the inset region,the inset display 310 displays a transitional portion of the image witha varying resolution. In other embodiments, peripheral display 315displays the transitional portion of the image. In yet otherembodiments, the transitional portion of the image is displayed on boththe inset display 310 and the peripheral display 315. The inset display310 receives image data associated with the inset portion from thecontroller 325. The inset display 310 emits image light of a firstpolarization that is folded by the MOA 320. In some embodiments, theimage light of the first polarization include circularly polarized lightof a first handedness, e.g., right handed circularly polarized light.

The peripheral display 315 displays the peripheral region portion of thecomposite content. The peripheral display 315 receives image dataassociated with the peripheral region from the controller 325. In someembodiments, the peripheral display 315 may support displaying only lowresolution content (e.g., it may be relatively low resolution display).In some embodiments, the peripheral display 315 may support displayingcontent at high resolution as well as low resolution content. In oneembodiment, an optic of the peripheral display 310 features an EFL ofapproximately 40 mm; a pixel pitch of a display surface of theperipheral display 315 is approximately 55 μm; and an instantaneous FOV(spatial resolution) is approximately 4.3 arcminute. The peripheraldisplay 315 emits image light at a second polarization, such that theimage light of the second polarization is not folded by the MOA 320. Insome embodiments, the image light of the second polarization includecircularly polarized light of a second handedness opposite to the firsthandedness, e.g., left handed circularly polarized light.

In some embodiments, the optical properties of the inset display 310 andthe peripheral display 315 are “well matched.” For example, a virtualimage distance of the inset display 310 and the peripheral display 310are within a threshold distance from each other. The threshold distancemay be determined by an amount of dioptric separation. The inset display310 and the peripheral display 315 are also well matched in the sensethat one or more aberrations (e.g., field curvature, astigmatism,longitudinal chromatic aberration, etc.) for both displays are within athreshold amount. If the inset display 310 and the peripheral display315 are not well matched, it may impede matching the virtual imagedistance for the inset display 310 and the peripheral display 315 whenthe inset display 310 is steered over the FOV.

In some embodiments, the display assembly 300 is configured to generatecomposite content having a fixed inset region. A fixed inset region isan inset region that is fixed in relation to the peripheral region. Thefixed inset region does not change its location with the movement of theeye. In some embodiments, the fixed inset region is located in an insetarea located in a center of the FOV, i.e., around an optical axis of theMOA 320. In other embodiments, the fixed inset region is located at someother location (e.g., may be off-center). In these embodiments, the MOA320 may also include an optical anti-aliasing filter. The opticalanti-aliasing filter is an optical element that optically blurs theperipheral region of the composite content. This helps remove digitalartifacts due to the nature of the display and make the blur of theperipheral region more natural.

The MOA 320 combines (multiplexes) the content from the inset display310 and the peripheral display 315 to form a composite content atretinal resolution, i.e., at a high resolution in a foveal region of theretina and at a lower resolution in the retinal region outside thefovea. The MOA 320 may include a pair of optical elements in opticalseries that form a pancake lens. The pancake lens of the MOA 320 maytransform image light of the first polarization emitted from the insetdisplay 310 into a first portion of image light having a thirdpolarization by folding the image light of the first polarization. Thepancake lens of the MOA 320 may further transform the image light of thesecond polarization emitted from the peripheral display 315 into asecond portion of image light having the third polarization by directlytransmitting (without folding) the image light of the secondpolarization. The MOA combines (multiplexes) the first portion of imagelight related to the high resolution inset region and the second portionof image light related to the lower resolution peripheral region intocomposite content. The MOA 320 directs the composite content towards aneye-box of the display assembly 300. More details about a structure andoperation of the MOA 320 are described in conjunction with FIG. 4.

In some embodiments, the controller 325 divides an image (or series ofimages) into a high resolution inset portion and a peripheral portion(lower resolution portion). Alternatively, the controller 325 divides animage (or series of images) into a high resolution inset portion, atransitional portion, and a peripheral portion. In some embodiments, thecontroller 325 adjusts the resolution (e.g., upsample or downsample) ofthe high resolution inset portion such that it corresponds to a targetresolution of an inset region. The target resolution is a resolutioncorresponding to a fovea region of a human eye. In some embodiments, thetarget resolution may be a resolution of the inset display 310. In someembodiments, the resolution of the high resolution inset portion is atthe resolution of the inset display 310 so no adjustment is needed. Theresulting content corresponds to the inset region of the compositecontent. Likewise, in some embodiments, the controller 325 adjusts(e.g., downsamples) the resolution of the peripheral portion such thatit corresponds to the resolution of a background region of the compositecontent (e.g., may be a resolution of the peripheral display 315). Theresulting content corresponds to the background region of the compositecontent.

In some embodiments, the controller 325 applies a blending function toadjust the resolution of transitional portion such that the resolutionsmoothly transitions from a resolution of the high resolution insetportion of the image to the resolution of the peripheral region. Theblending function corresponds to the fall off in acuity associated witha transition from a fovea to a non-fovea region of a human eye. Theblending function may be, for example, a Gaussian pyramid decompositionfunction, a Gaussian blending function, some function that smoothlytransitions from the resolution of the inset region to the resolution ofthe peripheral region, or some combination thereof. Additionally, thepyramid blending function may include performing a Gaussian pyramiddecomposition, i.e., smoothen the content with an appropriate smoothingfilter and then subsample the smoothed content and continue the processfor a predetermined level of sampling density. The sub sampled andsmoothened content is blended to the original content using a Gaussianblending function. The blended transitional portion corresponds to thetransitional region of the composite content.

The controller 325 may also fade (e.g. the light is reduced in thesection of the resulting image) the peripheral portion and/or thetransitional portion using an intensity fading function. The content mayinclude regions that have variable amounts of fading. Each region istermed as a fading region. The boundary of a fading region is determinedusing a size of the inset region. In some embodiments, the intensityfading function is applied to the image that causes an inset area in thebackground region to fade to black. And similarly, a different intensityfading function may be applied to some of the transitional portion ofthe image that surrounds the high resolution inset portion of the image.

The controller 325 provides, for display, the inset region to the insetdisplay 310. The controller 325 also provides, for display, theperipheral region to the peripheral display 315.

FIG. 4 shows a detailed view of cross section of the display assembly300 of FIG. 3 including the inset display 310, the peripheral display315, the MOA 320 and the controller 325, in accordance with anembodiment. The inset display 310 having a first resolution emits imagelight 405 of a first polarization. In some embodiments, the image light405 includes circularly polarized light of a first handedness, e.g., theimage light 405 is right handed circularly polarized light. In oneembodiment, the inset display 310 directly emits the image light 405 ascircularly polarized light. Alternatively, the inset display 310includes a display surface and a circular polarizer (not shown in FIG.4), and the display surface is configured to emit image light, and thecircular polarizer polarizes the image light to form the image light 405of the first polarization. The controller 325 is coupled to the insetdisplay 310 and the peripheral display 315, and the controller 325controls operations of the inset display 310 and the peripheral display315 as discussed in conjunction with FIG. 3. In some embodiments (notshown in FIG. 4), the controller 325 drives the inset display 310 andthe peripheral display 315 via a wired connection that goes from thecontroller 325 to the inset display 310 through a hole in the peripheraldisplay 315.

The peripheral display 315 having a second resolution different than thefirst resolution of the inset display 310 emits image light 410 of asecond polarization different than the first polarization. In someembodiments, the second resolution of the peripheral display 315 islower than the first resolution of the inset display 310. In someembodiments, the image light 410 includes circularly polarized light ofa second handedness opposite to the first handedness, e.g., the imagelight 410 is left handed circularly polarized light. In one embodiment,the peripheral display 315 directly emits the image light 410 ascircularly polarized light. Alternatively, the peripheral display 315includes a display surface and a circular polarizer (not shown in FIG.4), and the display surface is configured to emit image light, and thecircular polarizer polarizes the image light to form the image light 410of the second polarization.

As shown in FIG. 4, the MOA 320 is implemented as a pancake lensassembly that includes a front optical element 415 in optical serieswith a back optical element 420, wherein the front optical element 415is positioned closer to an eye-box 423. One or more surfaces of thefront optical element 415 and the back optical element 420 are shaped tocorrect for field curvature. One or more surfaces of the front opticalelement 415 may be shaped to be spherically concave (e.g., a portion ofa sphere), spherically convex, a rotationally symmetric sphere, afreeform shape, or some other shape that mitigates field curvature. Insome embodiments, the shape of one or more surfaces of the front opticalelement 415 and the back optical element 420 are designed toadditionally correct for other forms of optical aberration. In someembodiments, at least one of the front optical element 415 and the backoptical element 420 within the MOA 320 may have one or more coatings,such as anti-reflective coatings, e.g., to reduce ghost images andenhance contrast.

In some embodiments, the front optical element 415 and the back opticalelement 420 are separate lenses with an air gap between the frontoptical element 415 and the back optical element 420. In one or moreembodiments, the controller 325 is coupled to the MOA 320, i.e., to atleast one of the front optical element 415 and the back optical element420. For example, the controller 325 may be coupled to the front opticalelement 415 and/or the back optical element 420 via one or moremicro-actuators (positioners), not shown in FIG. 4. The controller 325may move the front optical element 415 and/or the back optical element420 (e.g., along z axis) via the one or more micro-actuators, based inpart on instructions from the controller 325. In this manner, thecontroller 325 may dynamically adjust the air gap between the frontoptical element 415 and the back optical element 420 in order to changea virtual image distance. Thus, the MOA 320 having the front opticalelement 415 and the back optical element 420 coupled to the controller325 may operate as a varifocal optical assembly. In other embodiments,the front optical element 415 and the back optical element 420 arebonded into a monolithic assembly. As shown in FIG. 4, a monolithic slab422 may bond the front optical element 415 and the back optical element420 without any air gap between the front optical element 415 and theback optical element 420. In yet other embodiments, inset display 310may be actuated along an optical axis of the MOA 320, e.g., along zdimension, in order to change a virtual image distance, providingvarifocal capability to the display assembly 300.

In one embodiment illustrated in FIG. 4, the back optical element 420includes a mirrored surface 425 and a waveplate surface 430. Themirrored surface 425 is partially reflective to reflect a portion oflight incident on the mirrored surface 425. In some embodiments, themirrored surface 425 is configured to transmit approximately 50% of theincident light and reflect approximately 50% of the incident light. Insome embodiments, the waveplate surface 430 is a quarter-waveplate thatshifts polarization of received light. A quarter-waveplate convertscircularly polarized light into linearly polarized light. Likewise, aquarter-waveplate can convert linearly polarized light incident to thequarter-waveplate into circularly polarized light. Quarter-waveplatescan be made of birefringent materials such as quartz, organic materialsheets, or liquid crystal.

In the embodiment shown in FIG. 4, the front optical element 415includes a reflective polarizer surface 435. The reflective polarizersurface 435 is a partially reflective mirror configured to reflectreceived light of a first linear polarization orthogonal to atransmission axis of the front optical element 415. The reflectivepolarizer surface 430 is also configured to transmit received light of asecond linear polarization parallel to the transmission axis of thefront optical element 415. For example, the reflective polarizer surface430 may be configured to reflect linearly polarized light with apolarization direction in the y direction, and pass light that islinearly polarized in the x direction. In some embodiments, thereflective polarizer surface 430 has a polarization transmissioncontrast ratio greater than 100 to 1, e.g., 200:1 or 500:1.

FIG. 4 further shows propagation of light in the display assembly 300that includes the inset display 310, the peripheral display 315 and theMOA 320, in accordance with an embodiment. A first portion of the imagelight 405 emitted from the inset display 310 is reflected by themirrored surface 425 of the back optical element 420, and a secondportion of the image light 405 is transmitted by the mirrored surface425 towards the waveplate surface 430. In some embodiments, the mirroredsurface 425 is configured to reflect approximately 50% of incident light(e.g., the image light 405). The waveplate surface 430(quarter-waveplate) has an axis 45 degrees (or 90 degrees) relative tothe x direction. The orientation of the waveplate axis relative to theincident circularly polarized light controls the polarization directionof the emitted linearly polarized light. Similarly, the orientation ofthe waveplate axis relative to the incident linearly polarized lightcontrols the handedness of the emitted circularly polarized light. Thewaveplate surface 430 changes polarization of the incident image light405 from circular polarization to linear polarization.

The linearly polarized image light 405 is incident on the reflectivepolarizer surface 435 of the front optical element 415, which reflectslight that is polarized in a blocking direction (e.g., y direction) andtransmits light that is polarized in a perpendicular direction (e.g., xdirection). At this point, the linearly polarized image light 405 islinearly polarized in the blocking direction. Thus, the reflectivepolarizer surface 435 reflects the linearly polarized image light 405.The waveplate surface 430 changes the reflected linearly polarized imagelight 405 back to the circularly polarized image light 405 having ahandedness identical to an initial handedness of the image light 405emitted from the inset display 310. The mirrored surface 425 reflects aportion of the circularly polarized image light 405, as described above.

The reflected portion of the circularly polarized image light 405 isalso circularly polarized; however, its handedness is opposite to theinitial handedness of the image light 405 due to the reflection from themirrored surface 425. Thus, the waveplate surface 430 changes thepolarization of the reflected portion of the circularly polarized imagelight 405 to linearly polarized image light 440. However, as thehandedness of the reflected portion of the circularly polarized imagelight 405 is opposite to the initial handedness of the image light 405emitted from the inset display 310, the image light 440 is linearlypolarized in a direction (e.g., x) perpendicular to the blockingdirection (e.g., y) and is therefore transmitted by the reflectivepolarizer surface 435 to the eye-box 423. The linearly polarized light440 may propagate through a center of curvature 445 before reaching atleast one surface of an eye, e.g., a foveal region of the eye (not shownin FIG. 4).

A first portion of the image light 410 emitted from the peripheraldisplay 315 is reflected by the mirrored surface 425, and a secondportion of the image light 410 is transmitted by the mirrored surface425 towards the waveplate surface 430. The waveplate surface 430 changespolarization of the incident image light 410 from circular polarizationto linear polarization. As a handedness of the circularly polarizedimage light 410 is opposite to the initial handedness of the circularlypolarized image light 405, the waveplate surface 430 transforms theimage light 410 into image light that is linearly polarized in adirection (e.g., x) perpendicular to the blocking direction (e.g., y)and is therefore transmitted by the reflective polarizer surface 435 tothe eye-box 423 as image light 450. The linearly polarized image light450 may propagate through the center of curvature 445 before reaching atleast one surface of the eye, e.g., a retinal region outside of thefovea of the eye (not shown in FIG. 4). Therefore, the MOA 320 combines(multiplexes) the image light 405, 410 of different polarization emittedfrom different resolution displays into image light of a linearpolarization. The higher resolution linearly polarized image light 440may illuminate the foveal region of the eye and the lower resolutionlinearly polarized image light 450 may illuminate the retinal regionoutside the fovea, thus achieving desired resolutions in each portion ofthe eye.

FIG. 4A shows an example propagation of image light 455 emitted from theinset display 310, in accordance with an embodiment. FIG. 4A showspropagation of the image light 455 through components of the MOA 320 fora particular field angle toward the eye-box 423. Note that the insetdisplay 310 may instantaneously cover one or more field angles.Typically, the inset display 310 covers a FOV between approximately −10degrees and +10 degrees.

FIG. 4B shows an example superposition of display paths from the insetdisplay 310 and the peripheral display 315, in accordance with anembodiment. Image light 460 of a first polarization is emitted from aparticular field angle of the inset display 310, and image light 465 ofa second polarization is emitted from the same field angle of theperipheral display 315. After propagating through components of the MOA320, as discussed in conjunction with FIG. 4, the image light 460 andthe image light 465 of different polarizations are superimposed intoimage light 470 of a single polarization directed to the eye-box 423.

FIG. 4C shows another example superposition of display paths from theinset display 310 and the peripheral display 315 for a field anglelarger than that in FIG. 4B, in accordance with an embodiment. Imagelight 475 of a first polarization and image light 480 of a secondpolarization is emitted from a larger field angle of the inset display310 than the image light 460 and the image light 465. After propagatingthrough components of the MOA 320, as discussed in conjunction with FIG.4, the image light 475 and the image light 480 are superimposed intoimage light 485 of a single polarization directed to the eye-box 423. Itshould be noted that the peripheral display 315 may be fully visible. Insome embodiments, the inset display 310 is vignettes away from theperipheral display 315.

FIG. 5 is a detailed view of cross section of the display assembly 300of FIG. 3 having the MOA 320 based on at least one birefringent element(lens), in accordance with an embodiment. As shown in FIG. 5, the MOA320 utilizes at least one birefringent element 505 in a stack within abirefringent lens block 510. The birefringent element 505 can be made ofcrystal, plastics, liquid crystals, etc. In an embodiment, thebirefringent element 505 may be implemented as a birefringent Fresnellens. The birefringent element 505 may be implemented to have a firstfocal length for a first polarization associated with image light 515,and a second focal length longer than the first focal length for asecond polarization orthogonal to the first polarization associated withimage light 520. The birefringent lens block 510 having the at least onebirefringent element 505 superimposes the image light 515 of the firstpolarization emitted from the inset display 310 and the image light 520of the second polarization emitted from the peripheral display 315 intoimage light 525 of a single polarization directed to an eye-box 530.

Note that a size of a bezel 535 of the inset display 310 can be designedto be small enough such that the bezel 535 would not shadow the imagelight 520 being at the same field of view as the image light 515 emittedfrom an edge of the inset display 310. In some embodiments, a portion ofthe peripheral display 315 positioned behind the inset display 310 isnot being used. The peripheral display 315 may be implemented as acustom display having a hole in a center of the peripheral display 315(not shown in FIG. 5). The hole in the center of the peripheral display315 may allow connection wires of a driving interface (not shown in FIG.5) to go through the peripheral display 315 and couple the peripheraldisplay 315 with the inset display 310.

In alternative embodiments, instead of the at least one birefringentelement 505, the MOA 320 may include at least one geometric phase lensor at least one Pancharatnam-Berry phase lens (not shown in FIG. 5). Thegeometric phase lens of the MOA 320 (or the Pancharatnam-Berry phaselens) may be implemented to have a focal length of f for image light ofa first polarization (e.g., left handed circularly polarized light), anda focal length of −f for image light of a second polarization (e.g.,right handed circularly polarized light). In some embodiments, thegeometric phase lens can be combined with other elements within the MOA320 having a focal length of f₀. Thus, the combined focal length for theMOA 320 having the geometric phase lens would be (f₀+f) for the imagelight of the first polarization (e.g., left handed circularly polarizedlight), and (f₀−f) for the image light of the second polarization (e.g.,right handed circularly polarized light). The rest of the systemfunctions similarly as what is described above for the system based onthe at least one birefringent element 505 of FIG. 5 and the system basedon a pancake lens of FIG. 4.

System Environment

FIG. 6 is a block diagram of one embodiment of a HMD system 600 in whicha console 610 operates. The HMD system 600 may operate in an artificialreality system. The HMD system 600 shown by FIG. 6 comprises a HMD 605and an input/output (I/O) interface 615 that is coupled to the console610. While FIG. 6 shows an example HMD system 600 including one HMD 605and on I/O interface 615, in other embodiments any number of thesecomponents may be included in the HMD system 600. For example, there maybe multiple HMDs 605 each having an associated I/O interface 615, witheach HMD 605 and I/O interface 615 communicating with the console 610.In alternative configurations, different and/or additional componentsmay be included in the HMD system 600. Additionally, functionalitydescribed in conjunction with one or more of the components shown inFIG. 6 may be distributed among the components in a different mannerthan described in conjunction with FIG. 6 in some embodiments. Forexample, some or all of the functionality of the console 610 is providedby the HMD 605.

The HMD 605 is a head-mounted display that presents content to a usercomprising virtual and/or augmented views of a physical, real-worldenvironment with computer-generated elements (e.g., two-dimensional (2D)or three-dimensional (3D) images, 2D or 3D video, sound, etc.). In someembodiments, the presented content includes audio that is presented viaan external device (e.g., speakers and/or headphones) that receivesaudio information from the HMD 605, the console 610, or both, andpresents audio data based on the audio information. The HMD 605 maycomprise one or more rigid bodies, which may be rigidly or non-rigidlycoupled together. A rigid coupling between rigid bodies causes thecoupled rigid bodies to act as a single rigid entity. In contrast, anon-rigid coupling between rigid bodies allows the rigid bodies to moverelative to each other. An embodiment of the HMD 605 may be the HMD 100described above in conjunction with FIG. 1.

The HMD 605 includes a DCA 620, an inset display 625, a peripheraldisplay 627, an optical assembly 630, one or more position sensors 635,an IMU 640, an optional eye tracking system 645, and an optionalvarifocal module 650. Some embodiments of the HMD 605 have differentcomponents than those described in conjunction with FIG. 6.Additionally, the functionality provided by various components describedin conjunction with FIG. 6 may be differently distributed among thecomponents of the HMD 605 in other embodiments.

The DCA 620 captures data describing depth information of a local areasurrounding some or all of the HMD 605. The DCA 620 can compute thedepth information using the data (e.g., based on a captured portion of astructured light pattern), or the DCA 620 can send this information toanother device such as the console 610 that can determine the depthinformation using the data from the DCA 620.

The inset display 625 is an electronic display that displaystwo-dimensional or three-dimensional images to the user in accordancewith data received from the console 610. In various embodiments, theinset display 625 comprises a single electronic display or multipleelectronic displays (e.g., a display for each eye of a user). The insetdisplay 625 is a high resolution display positioned along an opticalaxis of the optical assembly 630. The inset display 625 emits imagelight of a first polarization appropriately processed by the opticalassembly 630. In some embodiments, the inset display 625 emitscircularly polarized light of a first handedness, e.g., right handedcircularly polarized light. Examples of the inset display 625 include: aLCD, an OLED display, an ILED display, an AMOLED display, a TOLEDdisplay, some other display, or some combination thereof. The electronicdisplay 625 may be the embodiment of the inset display 310.

The peripheral display 627 is an electronic display that displaystwo-dimensional or three-dimensional images to the user in accordancewith data received from the console 610. In various embodiments, theperipheral display 627 comprises a single electronic display or multipleelectronic displays (e.g., a display for each eye of a user). Aresolution of the peripheral display 627 is typically lower than that ofthe inset display 625. The peripheral display 627 is positioned inparallel with the inset display 625, e.g., behind the inset display 625,covering a much wider FOV than the inset display 625. The peripheraldisplay 627 emits image light of a second polarization that isappropriately processed by the optical assembly 630. In someembodiments, the peripheral display 627 emits circularly polarized lightof a second handedness opposite to the first handedness. For example,the peripheral display 627 emits left handed circularly polarized light.Examples of the peripheral display 627 include: a LCD, an OLED display,an ILED display, an AMOLED display, a TOLED display, some other display,or some combination thereof. The peripheral display 627 may be theembodiment of the peripheral display 315.

The optical assembly 630 magnifies image light received from the insetdisplay 625 and the peripheral display 627, corrects optical errorsassociated with the image light, and presents the corrected image lightto a user of the HMD 605. The optical assembly 630 includes a pluralityof optical elements. Example optical elements included in the opticalassembly 630 include: an aperture, a Fresnel lens, a convex lens, aconcave lens, a filter, a reflecting surface, or any other suitableoptical element that affects image light. Moreover, the optical assembly630 may include combinations of different optical elements. In someembodiments, one or more of the optical elements in the optical assembly630 may have one or more coatings, such as partially reflective oranti-reflective coatings.

Magnification and focusing of the image light by the optical assembly630 allows the inset display 625 and the peripheral display 627 to bephysically smaller, weigh less and consume less power than largerdisplays. Additionally, magnification may increase the field-of-view ofthe content presented by the inset display 625 and the peripheraldisplay 627. For example, the field-of-view of the displayed content issuch that the displayed content is presented using almost all (e.g.,approximately 110 degrees diagonal), and in some cases all, of theuser's field-of-view. Additionally in some embodiments, the amount ofmagnification may be adjusted by adding or removing optical elements.

In some embodiments, the optical assembly 630 may be designed to correctone or more types of optical error. Examples of optical error includebarrel or pincushion distortions, longitudinal chromatic aberrations, ortransverse chromatic aberrations. Other types of optical errors mayfurther include spherical aberrations, chromatic aberrations or errorsdue to the lens field curvature, astigmatisms, or any other type ofoptical error. In some embodiments, content provided to the insetdisplay 625 and the peripheral display 627 for display is pre-distorted,and the optical assembly 630 corrects the distortion when it receivesimage light from the inset display 625 and the peripheral display 627generated based on the content.

In accordance with embodiments of the present disclosure, the opticalassembly 630 includes a MOA that combines (multiplexes) image lightemitted from the inset display 625 and the peripheral display 627. TheMOA of the optical assembly 630 may include a pair of optical elementsthat form a pancake lens assembly. The MOA of the optical assembly 630folds the image light of the first polarization emitted from the insetdisplay 625, and directly propagates the image light of the secondpolarization emitted from the peripheral display 627. Alternatively, theMOA of the optical assembly 630 includes at least one birefringent lensor at least one geometric phase lens. The MOA of the optical assembly630 directs the combined image light to an eye-box of a user. In someembodiments, the optical assembly 630 includes the MOA 320.

The IMU 640 is an electronic device that generates data indicating aposition of the HMD 605 based on measurement signals received from oneor more of the position sensors 635 and from depth information receivedfrom the DCA 620. A position sensor 635 generates one or moremeasurement signals in response to motion of the HMD 605. Examples ofposition sensors 635 include: one or more accelerometers, one or moregyroscopes, one or more magnetometers, another suitable type of sensorthat detects motion, a type of sensor used for error correction of theIMU 640, or some combination thereof. The position sensors 635 may belocated external to the IMU 640, internal to the IMU 640, or somecombination thereof.

Based on the one or more measurement signals from one or more positionsensors 635, the IMU 640 generates data indicating an estimated currentposition of the HMD 605 relative to an initial position of the HMD 605.For example, the position sensors 635 include multiple accelerometers tomeasure translational motion (forward/back, up/down, left/right) andmultiple gyroscopes to measure rotational motion (e.g., pitch, yaw,roll). In some embodiments, the position sensors 635 may represent theposition sensors 135 of FIG. 1. In some embodiments, the IMU 640 rapidlysamples the measurement signals and calculates the estimated currentposition of the HMD 605 from the sampled data. For example, the IMU 640integrates the measurement signals received from the accelerometers overtime to estimate a velocity vector and integrates the velocity vectorover time to determine an estimated current position of a referencepoint on the HMD 605. Alternatively, the IMU 640 provides the sampledmeasurement signals to the console 610, which interprets the data toreduce error. The reference point is a point that may be used todescribe the position of the HMD 605. The reference point may generallybe defined as a point in space or a position related to the HMD's 605orientation and position.

The IMU 640 receives one or more parameters from the console 610. Theone or more parameters are used to maintain tracking of the HMD 605.Based on a received parameter, the IMU 640 may adjust one or more IMUparameters (e.g., sample rate). In some embodiments, certain parameterscause the IMU 640 to update an initial position of the reference pointso it corresponds to a next position of the reference point. Updatingthe initial position of the reference point as the next calibratedposition of the reference point helps reduce accumulated errorassociated with the current position estimated the IMU 640. Theaccumulated error, also referred to as drift error, causes the estimatedposition of the reference point to “drift” away from the actual positionof the reference point over time. In some embodiments of the HMD 605,the IMU 640 may be a dedicated hardware component. In other embodiments,the IMU 640 may be a software component implemented in one or moreprocessors. In some embodiments, the IMU 640 may represent the IMU 130of FIG. 1.

In some embodiments, the eye tracking system 645 is integrated into theHMD 605. The eye tracking system 645 determines eye tracking informationassociated with an eye of a user wearing the HMD 605. The eye trackinginformation determined by the eye tracking system 645 may compriseinformation about an orientation of the user's eye, i.e., informationabout an angle of an eye-gaze. In some embodiments, the eye trackingsystem 645 is integrated into the optical assembly 630. An embodiment ofthe eye-tracking system 645 may comprise an illumination source and animaging device (camera).

In some embodiments, the varifocal module 650 is further integrated intothe HMD 605. The varifocal module 650 may be coupled to the eye trackingsystem 645 to obtain eye tracking information determined by the eyetracking system 645. The varifocal module 650 may be configured toadjust focus of one or more images displayed on the electronic display625, based on the determined eye tracking information obtained from theeye tracking system 645. In this way, the varifocal module 650 canmitigate vergence-accommodation conflict in relation to image light. Thevarifocal module 650 can be interfaced (e.g., either mechanically orelectrically) with at least one of the inset display 625, the peripheraldisplay 627, and at least one optical element of the optical assembly630. Then, the varifocal module 650 may be configured to adjust focus ofthe one or more images displayed on the inset display 625 and/or theperipheral display 627 by adjusting position of at least one of theinset display 625, the peripheral display 627 and the at least oneoptical element of the optical assembly 630, based on the determined eyetracking information obtained from the eye tracking system 645. Byadjusting the position, the varifocal module 650 varies focus of imagelight output from the inset display 625 and/or the peripheral display627 towards the user's eye. The varifocal module 650 may be alsoconfigured to adjust resolution of the images displayed on the insetdisplay 625 and/or the peripheral display 627 by performing foveatedrendering of the displayed images, based at least in part on thedetermined eye tracking information obtained from the eye trackingsystem 645. In this case, the varifocal module 650 provides appropriateimage signals to the inset display 625 and/or the peripheral display627. The varifocal module 650 provides image signals with a maximumpixel density for the inset display 625 only in a foveal region of theuser's eye-gaze, while providing image signals with lower pixeldensities in other regions of the inset display 625 and/or regions ofthe peripheral display 627. In one embodiment, the varifocal module 650may utilize the depth information obtained by the DCA 620 to, e.g.,generate content for presentation on the inset display 625 and/or theperipheral display 627.

The I/O interface 615 is a device that allows a user to send actionrequests and receive responses from the console 610. An action requestis a request to perform a particular action. For example, an actionrequest may be an instruction to start or end capture of image or videodata or an instruction to perform a particular action within anapplication. The I/O interface 615 may include one or more inputdevices. Example input devices include: a keyboard, a mouse, a gamecontroller, or any other suitable device for receiving action requestsand communicating the action requests to the console 610. An actionrequest received by the I/O interface 615 is communicated to the console610, which performs an action corresponding to the action request. Insome embodiments, the I/O interface 615 includes an IMU 640 thatcaptures IMU data indicating an estimated position of the I/O interface615 relative to an initial position of the I/O interface 615. In someembodiments, the I/O interface 615 may provide haptic feedback to theuser in accordance with instructions received from the console 610. Forexample, haptic feedback is provided when an action request is received,or the console 610 communicates instructions to the I/O interface 615causing the I/O interface 615 to generate haptic feedback when theconsole 610 performs an action.

The console 610 provides content to the HMD 605 for processing inaccordance with information received from one or more of: the DCA 620,the HMD 605, and the I/O interface 615. In the example shown in FIG. 6,the console 610 includes an application store 655, a tracking module660, and an engine 665. Some embodiments of the console 610 havedifferent modules or components than those described in conjunction withFIG. 6. Similarly, the functions further described below may bedistributed among components of the console 610 in a different mannerthan described in conjunction with FIG. 6.

The application store 655 stores one or more applications for executionby the console 610. An application is a group of instructions, that whenexecuted by a processor, generates content for presentation to the user.Content generated by an application may be in response to inputsreceived from the user via movement of the HMD 605 or the I/O interface615. Examples of applications include: gaming applications, conferencingapplications, video playback applications, or other suitableapplications.

The tracking module 660 calibrates the HMD system 600 using one or morecalibration parameters and may adjust one or more calibration parametersto reduce error in determination of the position of the HMD 605 or ofthe I/O interface 615. For example, the tracking module 660 communicatesa calibration parameter to the DCA 620 to adjust the focus of the DCA620 to more accurately determine positions of structured light elementscaptured by the DCA 620. Calibration performed by the tracking module660 also accounts for information received from the IMU 640 in the HMD605 and/or an IMU 640 included in the I/O interface 615. Additionally,if tracking of the HMD 605 is lost (e.g., the DCA 620 loses line ofsight of at least a threshold number of structured light elements), thetracking module 660 may re-calibrate some or all of the HMD system 600.

The tracking module 660 tracks movements of the HMD 605 or of the I/Ointerface 615 using information from the DCA 620, the one or moreposition sensors 635, the IMU 640 or some combination thereof. Forexample, the tracking module 650 determines a position of a referencepoint of the HMD 605 in a mapping of a local area based on informationfrom the HMD 605. The tracking module 660 may also determine positionsof the reference point of the HMD 605 or a reference point of the I/Ointerface 615 using data indicating a position of the HMD 605 from theIMU 640 or using data indicating a position of the I/O interface 615from an IMU 640 included in the I/O interface 615, respectively.Additionally, in some embodiments, the tracking module 660 may useportions of data indicating a position or the HMD 605 from the IMU 640as well as representations of the local area from the DCA 620 to predicta future location of the HMD 605. The tracking module 660 provides theestimated or predicted future position of the HMD 605 or the I/Ointerface 615 to the engine 655.

The engine 665 generates a 3D mapping of the area surrounding some orall of the HMD 605 (i.e., the “local area”) based on informationreceived from the HMD 605. In some embodiments, the engine 665determines depth information for the 3D mapping of the local area basedon information received from the DCA 620 that is relevant for techniquesused in computing depth. The engine 665 may calculate depth informationusing one or more techniques in computing depth from structured light.In various embodiments, the engine 665 uses the depth information to,e.g., update a model of the local area, and generate content based inpart on the updated model.

The engine 665 also executes applications within the HMD system 600 andreceives position information, acceleration information, velocityinformation, predicted future positions, or some combination thereof, ofthe HMD 605 from the tracking module 660. Based on the receivedinformation, the engine 665 determines content to provide to the HMD 605for presentation to the user. For example, if the received informationindicates that the user has looked to the left, the engine 665 generatescontent for the HMD 605 that mirrors the user's movement in a virtualenvironment or in an environment augmenting the local area withadditional content. Additionally, the engine 665 performs an actionwithin an application executing on the console 610 in response to anaction request received from the I/O interface 615 and provides feedbackto the user that the action was performed. The provided feedback may bevisual or audible feedback via the HMD 605 or haptic feedback via theI/O interface 615.

In some embodiments, based on the eye tracking information (e.g.,orientation of the user's eye) received from the eye tracking system645, the engine 665 determines resolution of the content provided to theHMD 605 for presentation to the user on the inset display 625 and/or theperipheral display 627. The engine 665 provides the content to the HMD605 having a maximum pixel resolution on the inset display 625 in afoveal region of the user's gaze, whereas the engine 665 provides alower pixel resolution in other regions of the inset display 625 and/orregions of the peripheral display 627, thus achieving less powerconsumption at the HMD 605 and saving computing cycles of the console610 without compromising a visual experience of the user. In someembodiments, the engine 665 can further use the eye tracking informationto adjust where objects are displayed on the inset display 625 and/orthe peripheral display 627 to prevent vergence-accommodation conflict.

Additional Configuration Information

The foregoing description of the embodiments of the disclosure has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the disclosure to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

Some portions of this description describe the embodiments of thedisclosure in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments of the disclosure may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a non-transitory, tangible computer readable storagemedium, or any type of media suitable for storing electronicinstructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the disclosure may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the disclosure be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thedisclosure, which is set forth in the following claims.

What is claimed is:
 1. A display assembly comprising: an inset displayhaving a first resolution configured to emit image light of a firstpolarization; a peripheral display having a second resolution configuredto emit image light of a second polarization; and a multiplexing opticalassembly (MOA) configured to: receive the image light of the firstpolarization and the image light of the second polarization, transformthe image light of the first polarization into a first portion of imagelight of a third polarization, transform the image light of the secondpolarization into a second portion of image light of the thirdpolarization, and direct the first portion of image light and the secondportion of image light toward an eye-box.
 2. The display assembly ofclaim 1, wherein the first resolution is higher than the secondresolution.
 3. The display assembly of claim 1, wherein the image lightof the first polarization and the image light of the second polarizationcomprise circularly polarized light of opposite handedness.
 4. Thedisplay assembly of claim 1, wherein the image light of the firstpolarization comprises right handed circularly polarized light and theimage light of the second polarization comprises left handed circularlypolarized light.
 5. The display assembly of claim 1, wherein the firstportion of image light of the third polarization and the second portionof image light of the third polarization includes light having apolarization parallel to a transmission axis of the MOA.
 6. The displayassembly of claim 1, wherein the inset display is positioned in parallelwith the peripheral display.
 7. The display assembly of claim 1, whereinthe MOA includes: a back optical element configured to: transmit adefined amount of light incident on a surface of the back opticalelement, and reflect a remaining amount of the light incident on thesurface of the back optical element; and a front optical element inoptical series with the back optical element and positioned closer tothe eye-box than the back optical element, the front optical elementconfigured to transmit toward the eye-box light received from the backoptical element having a polarization parallel to a transmission axis ofthe front optical element.
 8. The display assembly of claim 7, furthercomprising: a waveplate positioned between the front optical element andthe back optical element, the waveplate configured to: in-couplepolarized light, and transform the in-coupled polarized light intooutput light having different polarization than the in-coupled polarizedlight.
 9. The lens assembly of claim 8, wherein the waveplate is aquarter waveplate configured to transform the in-coupled light havingcircular polarization into the output light of a linear polarization.10. The lens assembly of claim 7, wherein the back optical element isconfigured to: transmit approximately 50% of the light incident on thesurface of the back optical element; and reflect approximately 50% ofthe light incident on the surface of the back optical element.
 11. Thedisplay assembly of claim 7, wherein: the back optical element isadjacent to the inset display; and the inset display emits the imagelight of the first polarization incident on the surface of the backoptical element.
 12. The display assembly of claim 1, wherein the insetdisplay is positioned along an optical axis of the MOA and the insetdisplay covers a first field-of-view (FOV) narrower than a second FOVcovered by the peripheral display.
 13. The display assembly of claim 1,wherein the MOA includes at least one birefringent element having afirst focal length for the image light of the first polarization and asecond focal length longer than the first focal length for the imagelight of the second polarization.
 14. The display assembly of claim 1,wherein the MOA includes at least one geometric phase lens having apositive focal length for the image light of the first polarization anda negative focal length for the image light of the second polarization.15. A head-mounted display (HMD) comprising: a display assemblyconfigured to emit image light, the display assembly including: an insetdisplay of a first resolution configured to emit image light of a firstpolarization, and a peripheral display of a second resolution configuredto emit image light of a second polarization; and a multiplexing opticalassembly (MOA) configured to: receive the image light of the firstpolarization and the image light of the second polarization, transformthe image light of the first polarization into a first portion of imagelight of a third polarization, transform the image light of the secondpolarization into a second portion of image light of the thirdpolarization, and direct the first portion of image light and the secondportion of image light toward an eye-box.
 16. The HMD of claim 15,wherein the image light of the first polarization and the image light ofthe second polarization comprise circularly polarized light of oppositehandedness.
 17. The HMD of claim 15, wherein the MOA includes: a backoptical element configured to: transmit a defined amount of lightincident on a surface of the back optical element, and reflect aremaining amount of the light incident on the surface of the backoptical element; a front optical element in optical series with the backoptical element and positioned closer to the eye-box than the backoptical element, the front optical element configured to transmit towardthe eye-box light received from the back optical element having apolarization parallel to a transmission axis of the front opticalelement; and a waveplate positioned between the front optical elementand the back optical element, the waveplate configured to: in-couplepolarized light, and transform the in-coupled polarized light intooutput light having different polarization than the in-coupled polarizedlight.
 18. The HMD of claim 15, wherein the MOA includes at least onebirefringent element having a first focal length for the image light ofthe first polarization and a second focal length longer than the firstfocal length for the image light of the second polarization.
 19. The HMDof claim 15, wherein the MOA includes at least one Pancharatnam-Berryphase lens having a positive focal length for the image light of thefirst polarization and a negative focal length for the image light ofthe second polarization.
 20. The HMD of claim 15, wherein the insetdisplay includes a display surface and a circular polarizer, and thedisplay surface is configured to emit image light, and the circularpolarizer polarizes the image light to form the image light of the firstpolarization.